Bias-variance tradeoff as darts

But the game of darts is more complicated

Two types of tradeoffs

Two types of tradeoffs

1. Explicit: Is a little bias worth the increase in precision?

2. Implicit: Improving precision without sacrificing unbiasedness?

Example

List experiment

Here is a list of things that some people have done.

List experiment

Please listen to them and then tell me HOW MANY of them you have done in the past two years.

List experiment

Do not tell me which ones. Just tell me HOW MANY:

List experiment

Do not tell me which ones. Just tell me HOW MANY:

Treatment group

1. Discussed politics with family or friends
2. Cast a ballot for governor Phil Bryant
3. Paid dues to a union
4. Given money to a Tea Party candidate

List experiment

Do not tell me which ones. Just tell me HOW MANY:

Treatment group

1. Discussed politics with family or friends
2. Cast a ballot for governor Phil Bryant
3. Paid dues to a union
4. Given money to a Tea Party candidate
5. Voted “YES” on the Personhood Initiative

Prevalence rate

\[ \text{Proportion(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]

Compare with direct question

Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?

Validation

Validation

Validation

Validation

Sensitive item

Organization X (advocating for immigration reduction and measures against undocumented immigration)

Three prevalence estimators

\[ \hat{\tau}_A = \text{Mean}(A_t) - \text{Mean}(A_c) \]

DLE yields more precise estimates

DLE yields more precise estimates

But variance reduction is not free

Different baseline estimates

Different baseline estimates

DLE variants

DLE variants

• Fixed-fixed is not an admissible design

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list
• Fixed-randomized and randomized-randomized shuffle sensitive item order

Carryover design effects

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Why does this happen?

Statistical tests

Statistical tests

• Goal: Detect asymmetric shift across treatment schedules

• Two tests:

1. Difference-in-differences (clustered responses)

2. Signed-rank test (paired responses)

Difference-in-differences test

\[ \hat{\tau}_1 = \text{Mean}(\text{First list}_t) - \text{Mean}(\text{First list}_c) \]

\[ \hat{\tau}_2 = \text{Mean}(\text{Second list}_t) - \text{Mean}(\text{Second list}_c) \]

Application to Alvarez et al (2019)

Application to Alvarez et al (2019)

Application to Alvarez et al (2019)

Background

Survey

• Facebook sample of Montevideo residents (N = 2688)

• Four criminal governance strategies

Survey

• Facebook sample of Montevideo residents (N = 2688)

• Four criminal governance strategies

DLE with placebo item

Things people have experienced in the last six months:

DLE with placebo item

Things people have experienced in the last six months:

DLE with placebo item

Things people have experienced in the last six months:

Prevalence estimates

Prevalence estimates

Prevalence estimates

Same for the other sensitive items!

What went wrong?

Assumptions

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Applying difference-in-differences test

Applying difference-in-differences test

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Additional:

• Known placebo proportion

  • No survey question

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Additional:

• Known placebo proportion

  • No survey question
  • No auxiliary information

Assumptions

Standard (Blair and Imai 2012):

• No liars (one-sided lying)

• No design effects

Additional:

• Known placebo proportion

  • No survey question
  • No auxiliary information
  • Construct bounds at different hypothetical values

DLE Estimate bounds

DLE Estimate bounds

DLE Estimate bounds

Conclusion

TBD

Appendix

• Alvarez et al (2019) details
• Montevideo survey details
• Signed rank test
• Cost simulations
• Performance simulations
• Placebo tests
• More bounds
• Network scale-up

Alvarez et al (2019) details

Mean supported organizations

Control list distributions

Montevideo survey

Stephenson’s signed rank test

\[ \underbrace{\widetilde{T} = \sum_{i=1}^N \text{sgn} \{(z_i - (1-z_i)) (Y_{i1} - Y_{i2})\} \times \tilde{q}_i}_{\text{Sum of signed ranks } \tilde{q}_i} \]